Systems and Methods for Tracking Objects

ABSTRACT

Various embodiments are disclosed for performing object tracking. One embodiment is a method for tracking an object in a plurality of frames, comprising obtaining a reference contour of an object in a reference frame and estimating, for a current frame after the reference frame, a contour of the object. The method further comprises comparing the reference contour with the estimated contour and determining at least one local region of the reference contour in the reference frame based on a difference between the reference contour and the estimated contour. Based on the difference, at least one corresponding region of the current frame is determined. The method further comprises computing a degree of similarity between the at least one corresponding region in the current frame and the at least one local region in the reference frame, adjusting the estimated contour in the current frame according to the degree of similarity, and designating the current frame as a new reference frame and a frame after the new reference as a new current frame.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to, and the benefit of, U.S.Provisional Patent Application entitled, “Systems and Methods forTracking Objects,” having Ser. No. 61/724,389, filed on Nov. 9, 2012,which is incorporated by reference in its entirety.

TECHNICAL FIELD

The present disclosure generally relates to video processing, and moreparticularly, to a system and method for tracking objects utilizing acontour weighting map.

BACKGROUND

Over the years, digital content has gained increasing popularity withconsumers. With the ever-growing amount of digital content available toconsumers through the Internet using computers, smart phones, and othersources, consumers have access to a vast amount of content. Furthermore,many devices (e.g., smartphones) and services are readily available thatallow consumers to capture and generate video content.

Upon capturing or downloading video content, the process of trackingobjects is commonly performed for editing purposes. For example, a usermay wish to augment a video with special effects where one or moregraphics are superimposed onto an object. In this regard, precisetracking of the object is important. However, challenges may arise whentracking objects, particularly as the object moves from frame to frame.This may cause, for example, the object to vary in shape and size.Additional challenges may arise when the object includes regions orelements that easily blend in with the background. This may be due tothe thickness of the elements, the color make-up of the elements, and/orother attributes of the elements.

SUMMARY

Briefly described, one embodiment, among others, is a method fortracking an object in a plurality of frames, comprising obtaining areference contour of an object in a reference frame and estimating, fora current frame after the reference frame, a contour of the object. Themethod further comprises comparing the reference contour with theestimated contour and determining at least one local region of thereference contour in the reference frame based on a difference betweenthe reference contour and the estimated contour. Based on thedifference, at least one corresponding region of the current frame isdetermined. The method further comprises computing a degree ofsimilarity between the at least one corresponding region in the currentframe and the at least one local region in the reference frame,adjusting the estimated contour in the current frame according to thedegree of similarity, and designating the current frame as a newreference frame and a frame after the new reference as a new currentframe.

Another embodiment is a system for tracking an object in a plurality offrames, comprising a processing device. The system further comprises anobject selector executable in the processing device for obtaining areference contour of an object in a reference frame and a contourestimator executable in the processing device for estimating, for acurrent frame after the reference frame, a contour of the object. Thesystem further comprises a local region analyzer executable in theprocessing device for: comparing the reference contour with theestimated contour, determining at least one local region of thereference contour in the reference frame based on a difference betweenthe reference contour and the estimated contour, determining at leastone corresponding region of the current frame based on the difference,and computing a degree of similarity between the at least onecorresponding region in the current frame and the at least one localregion in the reference frame. The contour estimator adjusts theestimated contour in the current frame according to the degree ofsimilarity and designates the current frame as a new reference frame anda frame after the new reference as a new current frame.

Another embodiment is a non-transitory computer-readable mediumembodying a program executable in a computing device, comprising codethat generates a user interface and obtains a reference contour of anobject in a reference frame, code that estimates, for a current frameafter the reference frame, a contour of the object, code that comparesthe reference contour with the estimated contour and code thatdetermines at least one local region of the reference contour in thereference frame based on a difference between the reference contour andthe estimated contour. The program further comprises code thatdetermines at least one corresponding region of the current frame basedon the difference, code that computes a degree of similarity between theat least one corresponding region in the current frame and the at leastone local region in the reference frame, code that adjusts the estimatedcontour in the current frame according to the degree of similarity, andcode that designates the current frame as a new reference frame and aframe after the new reference as a new current frame.

Other systems, methods, features, and advantages of the presentdisclosure will be or become apparent to one with skill in the art uponexamination of the following drawings and detailed description. It isintended that all such additional systems, methods, features, andadvantages be included within this description, be within the scope ofthe present disclosure, and be protected by the accompanying claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided by the Office upon request and paymentof the necessary fee.

Many aspects of the disclosure can be better understood with referenceto the following drawings. The components in the drawings are notnecessarily to scale, emphasis instead being placed upon clearlyillustrating the principles of the present disclosure. Moreover, in thedrawings, like reference numerals designate corresponding partsthroughout the several views.

FIG. 1 is a block diagram of a video editing system for facilitatingobject tracking in accordance with various embodiments of the presentdisclosure.

FIG. 2 is a detailed view of the video editing system device of FIG. 1in accordance with various embodiments of the present disclosure.

FIG. 3 is a top-level flowchart illustrating examples of functionalityimplemented as portions of the video editing system of FIG. 1 forfacilitating object tracking according to various embodiments of thepresent disclosure.

FIG. 4 depicts an example digital image to be processed by the videoediting system of FIG. 1 in accordance with various embodiments of thepresent disclosure.

FIG. 5 illustrates thin regions of an object to be tracked by the videoediting system of FIG. 1 in accordance with various embodiments of thepresent disclosure.

FIG. 6 illustrates the identification of local regions by the videoediting system of FIG. 1 in accordance with various embodiments of thepresent disclosure.

FIG. 7A illustrates selection of an object by a user using a selectiontool in a first frame.

FIGS. 7B-7E illustrate the object in succeeding frames.

FIG. 7F illustrates modification of the object based on the estimatedcontour.

FIG. 8 illustrates the refinement of an estimated contour performed bythe video editing system of FIG. 1 in accordance with variousembodiments of the present disclosure.

FIG. 9A illustrates an initial video frame or reference frame with anobject that the user wishes to track.

FIG. 9B illustrates a next frame in the video sequence.

FIG. 9C illustrates estimation of the direction of movement and themagnitude of movement in accordance with various embodiments of thepresent disclosure.

FIG. 9D illustrates a resulting object contour after the shape of theobject contour is modified in accordance with various embodiments of thepresent disclosure.

FIG. 9E illustrates an example where the estimated contour is missing aportion of the object.

FIG. 9F illustrates the result of a refined estimated contour inaccordance with various embodiments of the present disclosure.

FIG. 10A illustrates an initial video frame with an object that the userwishes to track.

FIG. 10B illustrates a next frame in the video sequence.

FIG. 10C illustrates an example of an estimated contour that erroneoulyincludes an additional region.

FIG. 10D illustrates identification of the additional region inaccordance with various embodiments of the present disclosure.

FIG. 10E illustrates the result of a refined estimated contour inaccordance with various embodiments of the present disclosure.

FIG. 11A illustrates an initial video frame and the object contour inputby the user.

FIG. 11B illustrates the next video frame, where local regions are usedfor refinement of the estimated contour in accordance with variousembodiments of the present disclosure.

FIGS. 11C and 11D illustrate an example of how the contour can changesubstantially due to partial occlusion of the tracked object by anindividual's hand in the frame.

FIG. 12A illustrates how the content close to a local region is shown asan pixel array for a video frame in accordance with various embodimentsof the present disclosure.

FIG. 12B illustrates the frame content for another video frame.

FIGS. 12C and 12D illustrate an example where the local regions cannotbe located precisely due to a small shift or deformation between thevideo frames or an error in the contour estimation.

FIGS. 12E and 12F illustrate how a measurement technique is utilized toevaluate local regions that are slightly misaligned while stillaccurately identifying local regions with a low degree of similarity inaccordance with various embodiments of the present disclosure.

DETAILED DESCRIPTION

The process of tracking one or more objects within a video stream may bechallenging, particularly when the object moves from frame to frame asthe object may vary in shape and size when moving from oneposition/location to another. Additional challenges may arise when theobject includes regions or elements that tend to blend in with thebackground. In order to produce high quality video editing results, anobject tracking system should accurately estimate the contour of theobject as the object moves. However, the object tracking process mayoccasionally yield erroneous results. For example, in some cases, one ormore portions of the object being tracked will not be completelysurrounded by the estimated contour that corresponds to an estimation ofwhere and how the object is positioned. As temporal dependency exists inthe object tracking process, an erroneous tracking result will, in manycases, lead to a series of erroneous results, thereby affecting videoediting process that follows.

In some cases, the user can reduce the number of erroneous results bymanually refining the estimated contour on a frame-by-frame basis asneeded and then allowing the tracking system to resume object trackingbased on the refinements made by the user. However, if a portion of theobject is difficult to track due to its color, shape, contour, or otherattributes, the object tracking algorithm may continually yielderroneous results for the portions of the object that are difficult totrack. This results in the user having to constantly refine the trackingresults in order to produce an accurate, estimated contour of theobject. This, of course, can be a time consuming process.

Various embodiments are disclosed for improving the tracking of objectswithin an input stream of frames, particularly for objects that includeelements or regions that may be difficult to track by conventionalsystems due to color, shape, contour, and other attributes. For someembodiments, the position and contour of the object is estimated on aframe-by-frame basis. The user selects a frame in the video and manuallyspecifies the contour of an object in the frame. As described in moredetail below, for the video frames that follow, the object trackingsystem iteratively performs a series of operations that include refiningestimated contours based on the contour in a previous frame.

First, an object contour in the current video frame is received from theuser and designated as a reference contour. An object tracking algorithmis then utilized to estimate the object contour in the next video frame,and a tracking result is generated whereby an estimated contour isderived. The object tracking system compares the generated trackingresult with the recorded reference contour, and a “local region”corresponding to a region containing the difference in contour betweenthe two is derived. Based on the content of the local region in thecurrent video frame and the content of the local region in the nextvideo frame, the object tracking system computes the similarity of thecorresponding local regions between the two video frames, and refinesthe tracking result (i.e., the estimated contour) of the next frameaccording to the degree of similarity. The iterative tracking processcontinues until all the frames are processed or until the user stops thetracking process.

A description of a system for facilitating object tracking is nowdescribed followed by a discussion of the operation of the componentswithin the system. FIG. 1 is a block diagram of a video editing system102 in which embodiments of the object tracking techniques disclosedherein may be implemented. The video editing system 102 may be embodied,for example, as a desktop computer, computer workstation, laptop, asmartphone 109, a tablet, or other computing platform that includes adisplay 104 and may include such input devices as a keyboard 106 and amouse 108.

For embodiments where the video editing system 102 is embodied as asmartphone 109 or tablet, the user may interface with the video editingsystem 102 via a touchscreen interface (not shown). In otherembodiments, the video editing system 102 may be embodied as a videogaming console 171, which includes a video game controller 172 forreceiving user preferences. For such embodiments, the video gamingconsole 171 may be connected to a television (not shown) or otherdisplay 104.

The video editing system 102 is configured to retrieve, via the mediainterface 112, digital media content 115 stored on a storage medium 120such as, by way of example and without limitation, a compact disc (CD)or a universal serial bus (USB) flash drive, wherein the digital mediacontent 115 may then be stored locally on a hard drive of the videoediting system 102. As one of ordinary skill will appreciate, thedigital media content 115 may be encoded in any of a number of formatsincluding, but not limited to, Motion Picture Experts Group (MPEG)-1,MPEG-2, MPEG-4, H.264, Third Generation Partnership Project (3GPP),3GPP-2, Standard-Definition Video (SD-Video), High-Definition Video(HD-Video), Digital Versatile Disc (DVD) multimedia, Video Compact Disc(VCD) multimedia, High-Definition Digital Versatile Disc (HD-DVD)multimedia, Digital Television Video/High-definition Digital Television(DTV/HDTV) multimedia, Audio Video Interleave (AVI), Digital Video (DV),QuickTime (QT) file, Windows Media Video (WMV), Advanced System Format(ASF), Real Media (RM), Flash Media (FLV), an MPEG Audio Layer III(MP3), an MPEG Audio Layer II (MP2), Waveform Audio Format (WAV),Windows Media Audio (WMA), or any number of other digital formats.

As depicted in FIG. 1, the media interface 112 in the video editingsystem 102 may also be configured to retrieve digital media content 115directly from a digital camera 107 where a cable 111 or some otherinterface may be used for coupling the digital camera 107 to the videoediting system 102. The video editing system 102 may support any one ofa number of common computer interfaces, such as, but not limited toIEEE-1394 High Performance Serial Bus (Firewire), USB, a serialconnection, and a parallel connection.

The digital camera 107 may also be coupled to the video editing system102 over a wireless connection or other communication path. The videoediting system 102 may be coupled to a network 118 such as, for example,the Internet, intranets, extranets, wide area networks (WANs), localarea networks (LANs), wired networks, wireless networks, or othersuitable networks, etc., or any combination of two or more suchnetworks. Through the network 118, the video editing system 102 mayreceive digital media content 115 from another computing system 103.Alternatively, the video editing system 102 may access one or more videosharing websites 134 hosted on a server 137 via the network 118 toretrieve digital media content 115.

The object selector 114 in the video editing system 102 is configured toobtain an object contour selection from the user of the video editingsystem 102, where the user is viewing and/or editing the media content115 obtained by the media interface 112. For some embodiments, theobjection selection is used as a reference contour where a local regionis derived for purposes of refining subsequent contour estimations, asdescribed in more detail below.

The contour estimator 116 is configured to estimate a contour on aframe-by-frame basis for the object being tracked. The local regionanalyzer 119 determines a local region based on a difference between thereference contour and the estimated contour. As referred to herein, a“local region” generally refers to one or more areas or regions within agiven frame corresponding to a portion or element of an object that islost or erroneously added during the tracking process. To furtherillustrate the concept of a local region, reference is made briefly toFIGS. 4-6, where FIG. 4 depicts an object 404 (i.e., a penguin) that auser wishes to track. As shown, the object 404 includes various elements(e.g., the flippers) which vary in size, shape, color, etc.

As shown in FIG. 5, the object 404 includes various elements or regionsthat blend in with the background, thereby resulting in “thin” regions502 a, 502 b due to the thin portions of the elements that are incontrast with the background of the image in the frame 402. As furtherillustrated in FIG. 6, the local regions 602 a, 602 b identified by thelocal region analyzer 119 comprises the portion of the object that islost (i.e., the flippers) during the tracking process. As described inmore detail below, these local regions 602 a, 602 b are analyzed acrossframes to further refine or correct the contour estimation derived bythe contour estimator 116. In some cases, the local regions 602 a, 602 bare added to an estimated contour in order to more accurately track theobject 404.

Turning now to FIG. 2, shown is a schematic diagram of the video editingsystem 102 shown in FIG. 1. The video editing system 102 may be embodiedin any one of a wide variety of wired and/or wireless computing devices,such as a desktop computer, portable computer, dedicated servercomputer, multiprocessor computing device, smartphone 109 (FIG. 1),tablet computing device, and so forth. As shown in FIG. 2, the videoediting system 102 comprises memory 214, a processing device 202, anumber of input/output interfaces 204, a network interface 206, adisplay 104, a peripheral interface 211, and mass storage 226, whereineach of these devices are connected across a local data bus 210.

The processing device 202 may include any custom made or commerciallyavailable processor, a central processing unit (CPU) or an auxiliaryprocessor among several processors associated with the video editingsystem 102, a semiconductor based microprocessor (in the form of amicrochip), a macroprocessor, one or more application specificintegrated circuits (ASICs), a plurality of suitably configured digitallogic gates, and other well known electrical configurations comprisingdiscrete elements both individually and in various combinations tocoordinate the overall operation of the computing system.

The memory 214 can include any one of a combination of volatile memoryelements (e.g., random-access memory (RAM, such as DRAM, and SRAM,etc.)) and nonvolatile memory elements (e.g., ROM, hard drive, tape,CDROM, etc.). The memory 214 typically comprises a native operatingsystem 217, one or more native applications, emulation systems, oremulated applications for any of a variety of operating systems and/oremulated hardware platforms, emulated operating systems, etc.

The applications may include application specific software which maycomprise some or all the components (media interface 112, objectselector 114, contour estimator 116, local region analyzer 119) of thevideo editing system 102 depicted in FIG. 1. In accordance with suchembodiments, the components are stored in memory 214 and executed by theprocessing device 202. One of ordinary skill in the art will appreciatethat the memory 214 can, and typically will, comprise other componentswhich have been omitted for purposes of brevity.

Input/output interfaces 204 provide any number of interfaces for theinput and output of data. For example, where the video editing system102 comprises a personal computer, these components may interface withone or more user input devices via the I/O interfaces 204, where theuser input devices may comprise a keyboard 106 (FIG. 1) or a mouse 108(FIG. 1). The display 104 may comprise a computer monitor, a plasmascreen for a PC, a liquid crystal display (LCD), a touchscreen display,or other display device 104.

In the context of this disclosure, a non-transitory computer-readablemedium stores programs for use by or in connection with an instructionexecution system, apparatus, or device. More specific examples of acomputer-readable medium may include by way of example and withoutlimitation: a portable computer diskette, a random access memory (RAM),a read-only memory (ROM), an erasable programmable read-only memory(EPROM, EEPROM, or Flash memory), and a portable compact disc read-onlymemory (CDROM) (optical).

With further reference to FIG. 2, network interface 206 comprisesvarious components used to transmit and/or receive data over a networkenvironment. For example, the network interface 206 may include a devicethat can communicate with both inputs and outputs, for instance, amodulator/demodulator (e.g., a modem), wireless (e.g., radio frequency(RF)) transceiver, a telephonic interface, a bridge, a router, networkcard, etc.). The video editing system 102 may communicate with one ormore computing devices via the network interface 206 over the network118 (FIG. 1). The video editing system 102 may further comprise massstorage 226. The peripheral interface 211 supports various interfacesincluding, but not limited to IEEE-1294 High Performance Serial Bus(Firewire), USB, a serial connection, and a parallel connection.

Reference is made to FIG. 3, which is a flowchart 300 in accordance withone embodiment for facilitating object tracking performed by the videoediting system 102 of FIG. 1. It is understood that the flowchart 300 ofFIG. 3 provides merely an example of the many different types offunctional arrangements that may be employed to implement the operationof the various components of the video editing system 102 (FIG. 1). Asan alternative, the flowchart of FIG. 3 may be viewed as depicting anexample of steps of a method implemented in the video editing system 102according to one or more embodiments.

Although the flowchart of FIG. 3 shows a specific order of execution, itis understood that the order of execution may differ from that which isdepicted. For example, the order of execution of two or more blocks maybe scrambled relative to the order shown. Also, two or more blocks shownin succession in FIG. 3 may be executed concurrently or with partialconcurrence. It is understood that all such variations are within thescope of the present disclosure.

Beginning with block 310, the object selector 114 (FIG. 1) in the videoediting system 102 obtains user input specifying a contour of an objectin a current frame. At this time, the frame serves as a current framefor the iterative tracking process. It may comprise the first frame in asequence of video frames or any frame in which the user selects as astarting point for tracking an object. The user may specify the contourthrough any number of selection or control means such as a paint brushtool on a user interface displayed to the user.

The user utilizes the region selection tool to specify or define thecontour of the object to be tracked in a video stream. After thetracking results are generated as described in more detail below, thetracking results may then be utilized for video editing. For example,the user may elect to adjust the color and/or brightness of the objector augment frames with the object with content from another videostream.

In block 315, the contour estimator 116 (FIG. 1) records the currentframe as the reference frame, and record the contour in the referenceframe as the reference contour. In block 320, select a frame after thereference frame as the current frame, and estimate a contour of theobject in the current frame. Note that the frame following the referenceframe is not limited to the frame immediately following the referenceframe and may comprise any frame following the reference frame (e.g.,the fifth frame following the reference frame). In this regard, theiterative tracking process involves processing the video sequence by oneor more frames during each iteration. Note that for the very firstiteration of the tracking process, the reference contour comprises thecontour defined by the user. However, for the iterations that follow,the reference contour comprises the refined contour of the next frameand so on.

In block 330, the local region analyzer 119 (FIG. 1) compares thereference contour with the estimated contour. That is, the contour ofthe reference frame is compared to the contour of the current frame suchthat contours spanning two successive frames are compared. Note,however, that the various embodiments disclosed are not limited to thecomparison of successive frames as the video editing system 102 may beconfigured to compare frames spaced farther apart.

In block 340, the local region analyzer 119 determines a local regionbased on a difference between the reference contour and the estimatedcontour. Referring back to the illustration of FIG. 6, the objectcontours for a reference frame (n) and the current frame (n+1) arecompared. As shown for this example, certain regions or elements 502 a,502 b (i.e., the flippers) are missing in frame (n+1). As a result, themissing regions are designated as local regions 602 a, 602 b forpurposes of refining the estimated object contour at frame (n+1).

With reference back to FIG. 3, in block 350, the local region analyzer119 computes a degree of similarity between a local region 602 a, 602 b(FIG. 6) in the reference frame (n) and a local region 602 a, 602 b inthe current frame (n+1). For some embodiments, the degree of similaritybetween local regions 602 a, 602 b in two frames may be calculated basedon a sum of absolute difference (SAD) metric between the pixels in thecorresponding local regions 602 a, 602 b. A low value of the sum ofabsolute difference indicates a large degree of similarity between thelocal regions 602 a, 602 b, and that the local regions 602 a, 602 b arealmost static across the two frames. Based on this, an inference can bemade that the object 404 itself has not moved significantly across thetwo frames.

The sum of absolute difference (SAD) metric used to compute the degreeof similarity is described in connection with FIGS. 12A-F. For a videoframe, the content close to a local region is shown as an pixel array inFIG. 12A, and the pixels in the local region 1202 are surrounded by thethick lines. The frame content for another video frame is shown in FIG.12B, where another local region 1204 is shown. For each pixel in a givenlocal region, there is a corresponding pixel in the other local region,where the corresponding pixel is identified according to the structureof the pixel array and based on the locations of the two regions in thevideo frames.

Determination of the SAD metric comprises computing the absolutedifference of pixel values for every pair of pixels and thenaccumulating the absolute differences as a measurement between the tworegions. A smaller SAD value indicates a higher similarity between tworegions, while a larger SAD value indicates that the two regions aredifferent. In the examples shown in FIG. 12A and FIG. 12B, only thetop-right pixels in the frames are different, and every pair of pixelsinside the local regions has the same value. This leads to a zero SADvalue, which denotes very high similarity between the local regions.

Many times during the tracking process, however, the local regionscannot be located precisely due, for example, to a small shift ordeformation between the video frames or an error in the contourestimation. An example of such a scenario is shown in FIG. 12C and FIG.12D, where the shape of the local region is the same as the previousexample, but where the location of local region 1208 has somedeformation in shape. Due to this small misalignment, the SAD valuecomputed based on the pixel pairs becomes significantly large, therebyerroneously indicating a small degree of similarity between the localregions.

Thus in accordance with various embodiments, a robust measurement isutilized where the SAD metric accurately evaluates local regions thatare slightly misaligned while still accurately identifying local regionswith a low degree of similarity. To achieve this, an alternative SADtechnique is implemented for various embodiments. With reference to FIG.12E and FIG. 12F, the SAD metric is computed based on pixel pairs. Forexample, a pixel A in local region 1206 is matched to a correspondingpixel in the other frame. For purposes of this disclosure, thecorresponding pixel in the other frame is referred to as an anchorpixel. The original SAD metric matches the pixel A to the anchor pixelA′, which leads to a large value of absolute difference. In contrast,the revised SAD metric performs a local search in a small range aroundthe anchor pixel A′ and identifies a pixel with the smallest absolutedifference. The small range in which the local search is performed maycomprise, for example, a pixel block (e.g., 3×3 or 5×5 pixel block wherethe anchor pixel is located at the center).

In the example shown, a local search reveals that pixel B′ has the samevalue as anchor pixel A′ and is therefore selected for purposes ofcomputing the absolute difference. A local search is performed for aplurality of pixel pairs to match a pixel in one frame to another pixelin the other frame. A reasonable range of the local search should besmall enough to identify the local regions with obviously differentcontent while also taking into account the misalignment of local regionsin one or two pixels. In this example, multiple searches are performedfor the regions 1206 and 1208 to compute their SAD value. Each searchyields a pixel pair from one region to the other region. Each localsearch may also select a pixel with a different position relative to theanchor pixel used for the search. For example, the selected pixel B isone pixel left to the anchor pixel A′, but the selected pixel in anothersearch may involve a pixel in a different position where the pixel isnot located one pixel left to the anchor pixel. This allows pixelmatching between two regions where slight deformation occurs, which istypical during video tracking.

Based on the disclosed local search mechanism, the final SAD value iscomputed based on the following formula:

SAD(R ₁ , R ₂)=Σ_(p) _(i) _(εP) ₁ min_(q) _(j) _(εS(anchor(p) _(i) ₎₎D(v(p _(i)), v(q _(j))),

where R₁, R₂ are the two regions, P₁ is a set of pixels which can be allpixels or a subset of pixels in R₁. For each pixel p_(i) in P₁,anchor(p_(i)) is the anchor pixel in the video frame containing R₂. Theanchor pixel corresponds to the pixel p_(i) and can be determined by thelocations of two regions in the video frames. S(anchor(p_(i)))represents a set of pixels as the search region according toanchor(p_(i)), and the search is performed for each pixel q_(j) in thesearch region. The values of pixel p_(i), q_(j) are represented asv(p_(i)), v(q_(j)), and D(v(p_(i)), v(q_(j))) is a metric for computingthe absolute difference of the values such that v(p_(i))={v_(i)(p_(i)),. . . , v_(n)(p_(i))}, v(q_(j)))={v₁(q_(j)), . . . , v_(n)(q_(j))}.

In various embodiments, each pixel contains a fixed number of channelsand there is a value for each channel. Each pixel contains at least onechannel with a value, wherein D(v(p_(i)), v(q_(j))) corresponds to theabsolute difference of the values according to one of the followingformulas:

D(v(p _(i)), v(q _(j)))=Σ_(k=1) ^(n) ∥v _(k)(p_(i))−v _(k)(q _(j))∥,

D(v(p _(i)), v(q _(j)))=Σ_(k=1) ^(n)(v _(k)(p _(i))−v _(k)(q _(j)))², or

D(v(p _(i)), v(q _(j))=√{square root over (Σ_(k=1) ^(n)(v _(k)(p _(i))−v_(k)(q _(j)))²)}{square root over (Σ_(k=1) ^(n)(v _(k)(p _(i))−v _(k)(q_(j)))²)},

where ∥x∥ is the absolute value of x. The metric corresponds tocomputing the absolute difference between the values of the two pixelsfor each channel and then accumulating the absolute differences amongall channels. However, in some cases, another metric may be usedrepresent the discrimination of pixel values, such as computing thesquare values of the differences and then accumulating the squaredvalues. The pixel q_(j) that contributes to the summation in SAD(R₁, R₂)is the pixel which results in the minimal absolute difference within thesearch region. By leveraging this revised SAD technique, the SAD valuecomputed from local regions 1206, 1208 is a relatively small value andindicates a high degree of similarity between the local regions 1206,1208.

Thus, when the local regions 602 a, 602 b are very similar across twoframes, an estimated contour with the local region(s) omitted willlikely be an erroneous estimate as the estimated contour differssubstantially from the previously estimated contour. In cases wherethere is not a large degree of similarity of the local regions 602 a,602 b across two frames, this typically means that the object has movedsignificantly or the shape of the object has changed substantiallybetween frames. For such cases, no further refinement is made to theestimated contour.

In block 360, based on the degree of similarity, the contour estimator116 adjusts or further refines the estimated contour. In cases wherethere is a large degree of similarity between the local regions 602 a,602 b across two frames and where the respective estimated contoursdiffer substantially (e.g., where one of the estimated contours ismissing the local region), the contour estimator 116 may be configuredto incorporate the missing local region(s) into the erroneous estimatedcontour as part of the refinement process.

To further illustrate the operations discussed above for blocks 350 and360, reference is made to FIG. 8, which illustrates estimated objectcontours across two frames (i.e., frame (n) and frame (n+1)). Asdescribed earlier in connection with FIG. 6, the local regions 602 a,602 b comprise the difference between the contours in the two frames. Inthe example of FIG. 8, there is a large degree of similarity between thelocal regions 602 a, 602 b across two frames and the respectiveestimated contours differ substantially (e.g., where one of theestimated contours is missing the local regions 602 a, 602 b). The largedegree of similarity between the local regions 602 a, 602 b may bedetermined based on a sum of absolute difference between pixels in thecorresponding local regions 602 a, 602 b. In this regard, a comparisonbetween pixel characteristics (e.g., pixel color) is performed on apixel-by-pixel basis between the local regions 602 a, 602 b in eachframe (frame (n) and frame (n+1)).

In the example of FIG. 8, there is a large degree of similarity betweenthe local regions 602 a, 602 b across two frames and the respectiveestimated contours differ substantially (e.g., where one of theestimated contours is missing the local region). In this case, thecontour estimator 116 (FIG. 1) may be configured to incorporate themissing local regions 602 a, 602 b into the erroneous estimated contourof frame (n+1) as part of the refinement process, as shown in FIG. 8.

At decision block 370, a determination is made on whether the last framein the video stream has been processed or whether the user wishes tostop the tracking process. If neither condition is true, the trackingprocess resumes back at block 315, where the contour estimation andlocal region comparison operations outlined in the blocks that followare repeated. Returning back to decision block 370, if at least one ofthe conditions is true, then the object tracking process stops, and theuser may then perform other operations via the video editing system 102,such as editing the tracked object based on the tracking results derivedin the remaining blocks above.

To further illustrate the various concepts disclosed, reference is madeto FIGS. 7 and 9-11, which illustrate various aspects of object trackingtechnique in accordance with various embodiments of the presentdisclosure. FIGS. 7A-F illustrate an example application of performingobject tracking. In FIG. 7A, the user selects or defines the contour ofthe object (i.e., the dog) using a selection tool such as brush tool asrepresented by the cursor tool shown.

The contour drawn around the object is represented by the outlinesurrounding the object. For the video frames that follow (as shown inFIGS. 7B-E), the object tracking algorithm estimates the contour of theobject on a frame-by-frame basis as the object moves and as the shape ofthe object changes. The object tracking results across the series offrames can then be utilized for editing purposes. As illustrated in FIG.7F, based on the estimated contour, the object may be modified (e.g.,color change) without modifying any of the other regions in the frame.In this regard, accurate object tracking is needed to facilitate videoediting operations.

Typically, the object being tracked moves or the shape of the objectchanges over time. However, the amount of movement tends to be fairlysmall within a short amount of time. Successive frames in a video aretypically spaced apart by approximately 1/30^(th) of a second. Thus,even if the object is moving or if the shape of the object changes, therate of change is relatively small on a frame-by-frame basis.

FIGS. 9A-F further illustrate the refinement operation of an estimatedcontour in accordance with various embodiments, where the differencebetween video frames is analyzed. FIG. 9A depicts an initial video frameor reference frame (frame (n)) with an object 902 that the user wishesto track. The bold line around the object 902 to be tracked representsan object contour 904 specified by the user using, for example, a paintbrush tool or other selection tool via a user interface displayed to theuser. Assume for purposes of illustration (and as shown in FIGS. 9B-F)that the object 902 moves in a downward direction towards the right.FIG. 9B depicts the next frame (e.g., frame (n+1)) in the videosequence. For every region within the object, the direction of movementand the magnitude of movement are estimated, as illustrated in FIG. 9C,where the arrows represent the direction and magnitude of movement bythe object.

Based on the information represented by the arrows in FIG. 9C, the shapeof the object contour 904 is warped or modified where the resultingobject contour 906 is shown in FIG. 9D. Note that for some embodiments,motion estimation may be performed on all the pixels in the entire frameand not just on those pixels within the object contour 904. For suchembodiments, the frame may be divided into blocks where motionestimation is then performed on each block.

Assume, for purposes of illustration, that the object tracking algorithmloses track of one or more portions/regions of the object 902. As shownin FIG. 9E, the estimated contour 907 is missing the tail and the feetof the tiger (the object 902 being tracked). In this scenario, themodified contour 906 in FIG. 9D rather than the initial contour 904 inFIG. 9A specified by the user is used as the reference contour in thecomparison for purposes of identifying the one or more local regions asthe modified contour 906 in FIG. 9D provides a better estimation of theobject shape in the next frame as it incorporates the difference betweenthe reference frame depicted in FIG. 9A and the current frame depictedin FIG. 9E. Moreover, the estimated movements can be used to shift thecorresponding local regions in the two frames in order to moreaccurately track the missing regions of the object (e.g., the tail andfeet of the tiger) more accurately. Note that at this moment, there arethree contours: 1) the original reference contour 904; 2) the modifiedcontour 906 derived based on motion estimation; and 3) the estimatedcontour 907 in the current frame derived by an arbitrary trackingalgorithm.

As shown in FIG. 9F, by supplementing the erroneous contour estimation907 in FIG. 9E with the local regions 908 a, 908 b, 908 c, 908 d, 908 eencompassing the tiger's tail and feet, a refined estimated contour 910including the local regions 908 a, 908 b, 908 c, 908 d, 908 e is derivedto provide a more accurate estimation object contour. For someembodiments, supplementing an erroneous contour estimation with thelocal region(s) comprises performing a union operation or determinationon the estimated contour and the local region to merge the two into alarger region.

Note that the refinement technique disclosed may also remove regionsthat are erroneously included in a contour estimation. Reference is madeto FIGS. 10A-E, which provide another example of the refinement of anestimated contour performed by the video editing system of FIG. 1 inaccordance with various embodiments of the present disclosure. FIG. 10Adepicts an initial video frame (frame (n)) with an object 1002 that theuser wishes to track. The bold line around the object 1002 to be trackedrepresents an object contour 1004 specified by the user using, forexample, a paint brush tool or other selection tool via a user interfacedisplayed to the user. Assume for purposes of illustration (and as shownin FIGS. 10B-E) that the object 902 moves in a downward directiontowards the right.

FIG. 10B depicts the next frame (e.g., frame (n+1)) in the videosequence. Again, for every region of the object, the direction ofmovement and the magnitude of movement are estimated. Based on motionestimation, the shape of the object contour 1004 is warped or modifiedwhere the resulting object contour 1006 is shown in FIG. 10B. Note thatfor some embodiments, motion estimation may be performed on all thepixels in the entire frame and not just on those pixels within theobject contour 1002. For such embodiments, the frame may be divided intoblocks where motion estimation is then performed on each block.

With reference to FIG. 10C, assume for the example shown that a region1007 is the tracking result for the frame, and a part of region 1008 iserroneously included the result that was not included in the estimatedcontour 1006. In this scenario, the refinement method identifies thisadditional region as a local region 1010 (FIG. 10D) and removes theerroneous region from the estimated contour to generate a refinedestimated contour, as shown in FIG. 10E. Note also that information frommotion estimation may be utilized to improve the accuracy in removingerroneous regions.

In some cases, certain restrictions may be implemented during the objecttracking process disclosed in order to further enhance the accuracy ofgenerating an estimated contour. For embodiments of the object trackingtechnique disclosed, a major assumption is that the previous trackingresult contains an accurate estimation of the contour. Based on thisassumption, the estimated contour may be further refined on aframe-by-frame basis.

Over time, however, the contour of the object may change substantially,thereby resulting in erroneous adjustments made based on an erroneouscontour. As such, comparison of other attributes other than the localregions may also be used, where such attributes include, for example,the color of the object and the color of the background. If the color ofthe region is close to the background color, then refining the estimatedcontour using this region may lead to an erroneous refinement due to thecolor of the local region matching the color of the background. As such,by utilizing other comparisons, the refinement process may be improved.

To further illustrate, reference is made to FIGS. 11A-D, whichillustrate how the object contour may change substantially over time.The initial video frame and the object contour 1102 input by the userare shown in FIG. 11A. FIG. 11B depicts the next video frame, where thetwo local regions 1106 a, 1106 b are used for refinement of theestimated contour 1104, as described herein. A comparison of the localregions of the two frames (FIGS. 11A and 11 B) reveals that the localregions have an intermediate similarity in the video frames. Also, thereference contour is exactly the original contour 1102 specified by theuser in FIG. 11A, and this implies the highest degree of agreementbetween the original contour and the reference contour. Accordingly,looser restrictions may be applied during the refinement process.

As shown in the example of FIGS. 11C and 11D, however, the contour canchange substantially due, for example, to partial occlusion of thetracked object by an individual's hand in the frame. A comparison of thelocal regions between the frames in FIGS. 11C and 11D reveals adifference represented by the region 1108 shown in FIG. 11D. However, acomparison of the reference contour 1110 in FIG. 11C with the originalcontour 1102 specified by the user in FIG. 11A reveals that the contour1110 has changed substantially over time during the tracking process. Inthis case, stricter restrictions may be applied to the threshold of thesimilarity in order to avoid erroneously refining the estimated contourusing regions that are not part of the tracked object (e.g., theindividual's hand). For the local region 1008, the similarity is nothigh enough to pass the stricter restrictions, so it will not be used torefine the contour.

For some embodiments, the original contour shape 1102 specified by theuser is compared to the reference contour 1110 by calculating a degreeof similarity between the original contour shape 1102 and the referencecontour 1110 to determine whether the two are substantially similar. Ifthe reference contour 1110 is substantially similar to the originalcontour 1102 specified by the user, then looser restrictions areapplied, otherwise stricter restrictions are applied.

It should be emphasized that the above-described embodiments of thepresent disclosure are merely possible examples of implementations setforth for a clear understanding of the principles of the disclosure.Many variations and modifications may be made to the above-describedembodiment(s) without departing substantially from the spirit andprinciples of the disclosure. All such modifications and variations areintended to be included herein within the scope of this disclosure andprotected by the following claims.

At least the following is claimed:
 1. A method implemented in an imageediting device for tracking an object in a plurality of frames,comprising: obtaining a reference contour of an object in a referenceframe; estimating, for a current frame after the reference frame, acontour of the object; comparing the reference contour with theestimated contour; determining at least one local region of thereference contour in the reference frame based on a difference betweenthe reference contour and the estimated contour; based on thedifference, determining at least one corresponding region of the currentframe; computing a degree of similarity between the at least onecorresponding region in the current frame and the at least one localregion in the reference frame; adjusting the estimated contour in thecurrent frame according to the degree of similarity; and designating thecurrent frame as a new reference frame and a frame after the newreference as a new current frame.
 2. The method of claim 1, wherein thereference contour is specified by a user via a user interface.
 3. Themethod of claim 1, wherein computing the degree of similarity betweenthe at least one corresponding region in the current frame and the atleast one local region in the reference frame is based on a sum ofabsolute difference between pixels in the current frame and pixels inthe reference frame based on the at least one local region and the atleast one corresponding region.
 4. The method of claim 3, whereincomputing the degree of similarity between the at least onecorresponding region in the current frame and the at least one localregion in the reference frame based on the sum of absolute differencebetween pixels comprises: determining a plurality of pixel pairs in theframes; computing absolute differences of the pixel values for the pixelpairs; and computing the sum of absolute difference value based onabsolute differences.
 5. The method of claim 4, wherein determining theplurality of pixel pairs in the frames is performed according to alocation of the local region in the reference frame and a location of acorresponding region in the current frame, each pixel pair comprising apixel in the current frame and a pixel in the reference frame.
 6. Themethod of claim 4, wherein computing the sum of absolute differencevalue comprises computing the sum of absolute difference of a region R₁and the other region R₂ based on the formula:SAD(R ₁ , R ₂)=Σ_(p) _(j) _(εP) ₁ min_(q) _(j) _(εS(anchor(p) _(i) ₎₎D(v(p _(i)), v(q _(j))), wherein P₁ is a subset of the pixels in theframe within R₁; p_(i) is a pixel in P₁; anchor(p_(i)) is the anchorpixel corresponding to pixel p_(i) and is located in the other framecontaining R₂; S(anchor(p_(i))) is a set of pixels determined by theanchor pixel; q_(j) is a pixel in S(anchor(p_(i))); v(p_(i)), v(q_(j))are the values of pixel p_(i), q_(j); and D(v(p_(i)), v(q_(j))) is ametric for computing the absolute difference of the values.
 7. Themethod of claim 6, wherein S(anchor(p_(i))) is a set of pixelsdetermined by anchor(p_(i)), wherein S(anchor(p_(i))) is the set ofpixels with a spatial distance to anchor(p_(i)) less than a predefinedthreshold.
 8. The method of claim 6, wherein D(v(p_(i)), v(q_(j)))correspond to a metric for computing the absolute difference of thevalues such that v(p_(i))={v₁(p_(i)), . . . , v_(n)(p_(i))},v(q_(j)))={v₁(q_(j)), . . . , v_(n)(q_(j))}, wherein each pixel containsat least one channel with a value, wherein D(v(p_(i)), v(q_(j)))corresponds to the absolute difference of the values calculatedaccording to one of the following formulas:D(v(p _(i)), v(q _(j)))=Σ_(k=1) ^(n) ∥v _(k)(p _(i))−v _(k)(q _(j))∥,D(v(p _(i)), v(q _(j)))=Σ_(k=1) ^(n)(v _(k)(p _(i))−v _(k)(q _(j)))², orD(v(p _(i)), v(q _(j)))=√{square root over (Σ_(k=1) ^(n)(v _(k)(p_(i))−v _(k)(q _(j)))²)}{square root over (Σ_(k=1) ^(n)(v _(k)(p _(i))−v_(k)(q _(j)))²)}, wherein ∥x∥ is the absolute value of x.
 9. The methodof claim 1, wherein adjusting the estimated contour in the current frameaccording to the degree of similarity comprises: in response to thedegree of similarity being greater than a threshold and in response tothe difference between the reference contour and the reference contourcomprising at least one region not included in the estimated contour,including the at least one local region of the current frame as part ofthe estimated contour for the current frame.
 10. The method of claim 1,wherein adjusting the estimated contour in the current frame accordingto the degree of similarity comprises: in response to the degree ofsimilarity being greater than a threshold and in response to thedifference between the reference contour and the estimated contourcomprising at least one region not included in the reference contour,removing the at least one local region from the estimated contour of thecurrent frame.
 11. The method of claim 1, wherein comparing thereference contour with the estimated contour comprises: performingmotion estimation on pixels within the reference contour in thereference frame; modifying the reference contour based on the motionestimation; and using the modified reference contour in the comparisonwith the estimated contour.
 12. The method of claim 1, wherein anoriginal contour of the object is obtained, and adjusting the estimatedcontour in the current frame further comprises: computing a degree ofagreement between the original contour and the reference contour; andadjusting the estimated contour according to the degree of similaritybetween the regions and the degree of agreement between the contours.13. The method of claim 12, wherein the original contour is obtained bya user via a user interface.
 14. The method of claim 12, wherein thedegree of agreement between the original contour and the referencecontour is computed according to the area of overlapped region for thecontours.
 15. The method of claim 12, wherein adjusting the estimatedcontour is performed in response to the degree of similarity between theregions being greater than a threshold, and wherein the threshold isdetermined according to the degree of agreement between the contourssuch that a lower threshold is set for a higher degree of agreement anda higher threshold is set for a lower degree of agreement.
 16. Themethod of claim 1, wherein all the steps are repeated until at least oneof the following conditions is met: a last frame in the plurality offrames is processed; and the user halts the tracking process.
 17. Asystem for tracking an object in a plurality of frames, comprising: aprocessing device; an object selector executable in the processingdevice for obtaining a reference contour of an object in a referenceframe; a contour estimator executable in the processing device forestimating, for a current frame after the reference frame, a contour ofthe object; a local region analyzer executable in the processing devicefor: comparing the reference contour with the estimated contour;determining at least one local region of the reference contour in thereference frame based on a difference between the reference contour andthe estimated contour; determining at least one corresponding region ofthe current frame based on the difference; and computing a degree ofsimilarity between the at least one corresponding region in the currentframe and the at least one local region in the reference frame, whereinthe contour estimator adjusts the estimated contour in the current frameaccording to the degree of similarity and designates the current frameas a new reference frame and a frame after the new reference as a newcurrent frame.
 18. The system of claim 17, wherein the reference contouris specified by a user via a user interface.
 19. The system of claim 17,wherein computing the degree of similarity between the at least onecorresponding region in the current frame and the at least one localregion in the reference frame is based on a sum of absolute differencebetween pixels in the current frame and pixels in the reference framebased on the at least one corresponding region and the at least onelocal region.
 20. The system of claim 19, wherein computing the degreeof similarity between the at least one corresponding region in thecurrent frame and the at least one local region in the reference framebased on the sum of absolute difference between pixels comprises:determining a plurality of pixel pairs in the frames; computing absolutedifferences of the pixel values for the pixel pairs; and computing thesum of absolute difference value based on absolute differences.
 21. Thesystem of claim 20, wherein determining the plurality of pixel pairs inthe frames is performed according to a location of the local region inthe reference frame and a location of a corresponding region in thecurrent frame, each pixel pair comprising a pixel in the current frameand a pixel in the reference frame.
 22. The system of claim 20, whereincomputing the sum of absolute difference value comprises computing thesum of absolute difference of a region R₁ and the other region R₂ basedon the formula:SAD(R ₁ , R ₂)=Σ_(p) _(j) _(εP) ₁ min_(q) _(j) _(εS(anchor(p) _(i) ₎₎D(v(p _(i)), v(q _(j)), wherein P₁ is a subset of the pixels in theframe within R₁; p_(i) is a pixel in P₁; anchor(p_(i)) is the anchorpixel corresponding to pixel p_(i) and is located in the other framecontaining R₂; S(anchor(p_(i))) is a set of pixels determined by theanchor pixel; q_(j) is a pixel in S(anchor(p_(i))); v(p_(i)), v(q_(j))are the values of pixel p_(i), q_(j); and D(v(p_(i)), v(q_(j))) is ametric for computing the absolute difference of the values.
 23. Thesystem of claim 17, wherein adjusting, by the contour estimator, theestimated contour in the current frame according to the degree ofsimilarity comprises: in response to the degree of similarity beinggreater than a threshold and in response to the difference between thereference contour and the reference contour comprising at least oneregion not included in the estimated contour, including the at least onelocal region of the current frame as part of the estimated contour forthe current frame.
 24. The system of claim 17, wherein adjusting, by thecontour estimator, the estimated contour in the current frame accordingto the degree of similarity comprises: in response to the degree ofsimilarity being greater than a threshold and in response to thedifference between the reference contour and the estimated contourcomprising at least one region not included in the reference contour,removing the at least one local region from the estimated contour of thecurrent frame.
 25. The system of claim 17, wherein comparing thereference contour with the estimated contour comprises: performingmotion estimation on pixels within the reference contour in thereference frame; modifying the reference contour based on the motionestimation; and using the modified reference contour in the comparisonwith the estimated contour.
 26. The method of claim 17, wherein anoriginal contour of the object is obtained, and adjusting the estimatedcontour in the current frame further comprises: computing a degree ofagreement between the original contour and the reference contour; andadjusting the estimated contour according to the degree of similaritybetween the regions and the degree of agreement between the contours.27. The method of claim 26, wherein the original contour is obtained bya user via a user interface.
 28. The method of claim 26, wherein thedegree of agreement between the original contour and the referencecontour is computed according to the area of overlapped region for thecontours.
 29. The method of claim 26, wherein adjusting the estimatedcontour is performed in response to the degree of similarity between theregions being greater than a threshold, and wherein the threshold isdetermined according to the degree of agreement between the contourssuch that a lower threshold is set for a higher degree of agreement anda higher threshold is set for a lower degree of agreement.
 30. Anon-transitory computer-readable medium embodying a program executablein a computing device, comprising: code that generates a user interfaceand obtains a reference contour of an object in a reference frame; codethat estimates, for a current frame after the reference frame, a contourof the object; code that compares the reference contour with theestimated contour; code that determines at least one local region of thereference contour in the reference frame based on a difference betweenthe reference contour and the estimated contour; code that determines atleast one corresponding region of the current frame based on thedifference; code that computes a degree of similarity between the atleast one corresponding region in the current frame and the at least onelocal region in the reference frame; code that adjusts the estimatedcontour in the current frame according to the degree of similarity; andcode that designates the current frame as a new reference frame and aframe after the new reference as a new current frame.
 31. Thenon-transitory computer-readable medium of claim 30, wherein the codethat computes the degree of similarity between the at least onecorresponding region in the current frame and the at least one localregion in the reference frame computes the degree of similarity based ona sum of absolute difference between pixels in the current frame andpixels in the reference frame based on the at least one correspondingregion and the at least one local region.
 32. The non-transitorycomputer-readable medium of claim 30, wherein the code that adjusts theestimated contour in the current frame according to the degree ofsimilarity includes the at least one local region of the current frameas part of the estimated contour for the current frame in response tothe degree of similarity being greater than a threshold and in responseto the difference between the reference contour and the referencecontour comprising at least one region not included in the estimatedcontour.
 33. The non-transitory computer-readable medium of claim 30,wherein the code that adjusts the estimated contour in the current frameaccording to the degree of similarity removes the at least one localregion from the estimated contour of the current frame in response tothe degree of similarity being greater than a threshold and in responseto the difference between the reference contour and the estimatedcontour comprising at least one region not included in the referencecontour.
 34. The non-transitory computer-readable medium of claim 30,wherein the code that compares the reference contour with the estimatedcontour performs motion estimation on pixels within the referencecontour in the reference frame further modifies the reference contourbased on the motion estimation and uses the modified reference contourin the comparison with the estimated contour.
 35. The non-transitorycomputer-readable medium of claim 30, wherein an original contour of theobject is obtained, and adjusting the estimated contour in the currentframe further comprises: computing a degree of agreement between theoriginal contour and the reference contour; and adjusting the estimatedcontour according to the degree of similarity between the regions andthe degree of agreement between the contours.
 36. The non-transitorycomputer-readable medium of claim 35, wherein adjusting the estimatedcontour is performed in response to the degree of similarity between theregions being greater than a threshold, and wherein the threshold isdetermined according to the degree of agreement between the contourssuch that a lower threshold is set for a higher degree of agreement anda higher threshold is set for a lower degree of agreement.